feat(gateway): safely rotate ACME credentials - #935

Problem

A corrupt persisted ACME credential must not take down Gateway or discard certificates that are already serving traffic. Operators also need a controlled way to replace the shared ACME account and its account-bound CAA records — including the documented staging→production directory switch, where the previous behavior silently registered a fresh account while every domain's CAA stayed pinned to the old one, blocking all issuance.

Solution

Fail closed on corrupt credentials — at both layers. A stored credential record that no longer decodes (WaveKV MessagePack layer) or whose embedded JSON is malformed fails the affected issuance/renewal attempt with an error pointing at the rotation RPC, without affecting Gateway startup, TLS serving, or other certificate state. get_acme_credentials distinguishes a missing/deleted key (None) from corruption (Err); treating corruption as absence would silently register a fresh account that the account-bound CAA records refuse. The previous silent re-registration was not a real self-heal for the same reason.

Admin-only RotateAcmeCredentials RPC, sequenced so that failures are convergent:

1. Validate every ZT domain's DNS credential up front (constructing the DNS client resolves the zone through an authenticated API call). A misconfigured domain aborts here with zero side effects and no rate-limited ACME registration consumed.
2. Register the replacement account.
3. Publish the replacement credential to WaveKV immediately. Nodes create an ACME client per operation, so they pick it up on their next attempt after synchronization.
4. Re-pin issue/issuewild CAA records for every ZT domain, best effort across all domains; the first domain reuses the registration client. Failed domains are named in the returned error together with the recovery path.
5. Attest the new account last: attestation does not gate issuance, so its agent round trips must not widen the publish→re-pin window. It runs even on partial re-pin failure so the new account is recorded.

Publishing before re-pinning makes partial failure recoverable without registering another account: rerun SetCaa (idempotent) until it succeeds. With the reverse order, every retry would burn one of Let's Encrypt's 10 new registrations per IP per 3 hours and the already-re-pinned domains would forbid the still-published old account.

Cross-node serialization via a best-effort TTL lock in WaveKV, alongside the in-process guard shared with CAA reconciliation. WaveKV has no CAS, so the lock narrows the concurrent-rotation window to replication latency rather than guaranteeing mutual exclusion; operators should still rotate through one gateway at a time. Release is guarded: a rotation that outlived the timeout does not delete the lock of the node that took over.

ACME URL mismatch now refuses to auto-register and points at RotateAcmeCredentials, which re-pins CAA along with the switch. The gateway setup tutorial's staging→production step now includes the rotation call.

AcmeInfo.account_uri derives from the published credentials instead of the best-effort attestation record, so the dashboard reflects a rotation even when quote generation is unavailable; the attestation quote is attached only when it matches the current account.

Private ACME credential material is never returned through the API.

Known limitations

• Rotation re-pins issuance to the new account; it does not deactivate the old ACME account at the CA (gateway: ACME rotation does not deactivate the old account at the CA #1009). Not a substitute for compromise response where the attacker controls DNS.
• The domain list is the local WaveKV snapshot; a ZT domain added on another node but not yet synced is not re-pinned until a later SetCaa run (gateway: no automatic CAA reconciliation loop #1010).
• The rotation lock is best-effort by construction (gateway: concurrent ACME rotation is only best-effort serialized #1008).
• CAA re-pinning briefly installs ; guard records; a failure can leave a domain blocked from issuance until a SetCaa rerun succeeds (same pre-existing hazard as SetCaa itself, documented on both paths).

Verification

• cargo test -p dstack-gateway — 77 passed (distributed_certbot: rotation lock/precondition cases, stale-holder release guard, URL matching, CAA reconciliation guards; kv::acme_credentials_tests: corrupt-record fail-closed, tombstone vs corruption)
• cargo clippy -- -D warnings -D clippy::expect_used -D clippy::unwrap_used --allow unused_variables — clean (workspace, CI flags)
• cargo fmt --all -- --check, git diff --check